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Abstract 

The second-order achievable rate region in Slepian-Wolf source coding systems is determined. 
The concept of second-order achievable rates, which enables us to make a finer evaluation of 
achievable rates, has already been introduced and analyzed for general sources in the single-user 
source coding problem. Analogously, in this paper, we first define the second-order achievable 
rate region for the Slepian-Wolf coding system and establish the source coding theorem in the 
second-order sense. The Slepian-Wolf coding problem for correlated sources is one of the typical 
problems in the multi-terminal information theory. In particular, Miyake and Kanaya, and Han 
have established the first- order sonvcc coding theorems for genera/ correlated sources. On the other 
hand, the second-order achievable rate problem for the Slepian-Wolf coding system remains still 
open up to present. In this paper we present the analysis concerning the second- order achievable 
rates for general sources which are on the basis of the information spectrum methods developed 
by Han and Verdii. Moreover, we compute the explicit second-order achievable rate region for 
i.i.d. correlated sources and mixed correlated sources respectively, using the relevant asymptotic 
normality. 
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I. Introduction 

We establish the second-order source coding theorems for the Slepian-Wolf coding system. In 
the single-source coding system, Han and Verdii [Ij and Steinberg and Verdii [2] have shown the 
source coding theorem for general sources using the information spectrum methods. Since the 
class of general sources is quite large, their results are very fundamental and useful. On the other 
hand, there are several researches concerning a finer evaluation of achievable rates called rates of 
the second-order. In variable-length source coding, Kontoyiannis [3] has established the second- 
order source coding theorem. In channel coding problems, Strassen (see, Csiszar and Korner [1]), 
Hayashi [5], Polyanskiy, Poor and Verdii [6], have determined the second-order capacity. In addition, 
Hayashi [7] has given the second-order optimal achievability theorem for the fixed-length source 
coding problem for general sources, and actually computed it for i.i.d. sources by invoking the 
asymptotic normality. From his analyses we know that the information spectrum methods [8] are 
still effective also for the evaluation on second-order achievable rates. Nomura and Han [9] have 
computed the second-order optimal rate for mixed sources, which is a typical case of nonergodic 
source, again by invoking the relevant asymptotic normality. 

In the area of multi-source coding problems, there are various types of source coding problems 
|10] . The Slepian-Wolf coding problem for two correlated sources is one of such typical problems 
|10j . In particular, there are two typical settings in the Slepian-Wolf coding problem. One is 
the setting that the decoder decodes only the sequence emitted from the one source, and another 
sequence emitted from another source is used as side-information. The other is the setting that 
the decoder have to decode both of two sequences emitted from two correlated sources. We call 
the former, the full side-information type problem in this paper. 

In the full side-information type problem, the first-order achievable rates for general correlated 
sources have been determined by Steinberg and Verdii [12]. On the other hand, Watanabe, Mat- 
sumoto and Uyematsu |13j . and Nomura and Matsushima |14j have considered the second-order 
achievable rate problem with full side-information; in this special case, Watanabe, Matsumoto and 
Uyematsu [13], have given only a "necessary" condition on the second-order achievable rates for 
i.i.d. correlated sources, while Nomura and Matsushima |14j have exactly determined the second- 
order achievable rates for i.i.d. correlated sources. Their results hold also on the basis of information 
spectrum methods and the asymptotic normality. 

In this paper, we consider the ordinary Slepian-Wolf coding problem without full side-information. 
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that is, the decoder have to reconstruct both of source sequences. In this setting, Miyake and 
Kanaya |15j with finite source alphabets, as well as Han [Sj with countably infinite source alphabets 
has established the first-order source coding theorem for general correlated sources. The second- 
order achievable rate region without full side-information for the Slepian-Wolf coding problem 
was first considered by Nomura and Matsushima |16j : they have focused only on i.i.d. correlated 
sources, where, unfortunately, the sufficient condition and the necessary condition do not coincide. 
On the other hand, in this paper, we shall determine the necessary and sufficient condition for 
general correlated sources. Furthermore, for i.i.d. correlated sources and mixed correlated sources, 
we apply our fundamental results to derive the explicit second-order achievable rate region. In the 
second-order analysis for i.i.d. correlated sources, the multivariate normal distribution function due 
to the central limit theorem plays the key role, while in the previous researches on the second- 
order achievable rates (cf. [9], [TB], [14j . |16j). only one-dimensional normal distribution function 
was enough to consider. Recently, Tan and Kosut pT] have independently established the second- 
order source coding theorem for i.i.d. correlated sources, which is derived as a special case of the 
main results in this paper. 

The analyses here are also on the basis of information spectrum methods invoking multi-dimensional 
distribution functions. In the second-order analysis for mixed correlated sources, we extend our 
results for i.i.d. correlated sources to the mixed correlated sources consisting of i.i.d. correlated 
sources. The class of mixed correlated sources is very important, because all of stationary sources 
can be regarded as forming mixed sources obtained by mixing stationary ergodic sources with 
respect to appropriate probability measure. The first-order achievable rate region for mixed corre- 
lated sources has already been computed by Han [8j on the basis of the information spectrum 
methods. We compute the second-order achievable rate region also by using the information 
spectrum methods. 

This paper is organized as follows. In Section II, we define the general correlated sources and 
achievable rate region. Then, we review the previous results on the first-order achievable rate region. 
In Section III, we derive the second-order achievable rate region for general sources by invoking 
the information spectrum methods. In Section IV we compute the second-order achievable rate 
region for i.i.d. correlated sources by using the asymptotic normality. In Section V, we compute 
the second-order achievable rate region for mixed correlated sources, again by using the relevant 
asymptotic normality. Finally, we conclude our results in Section VI. 
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II. Preliminaries 

A. Correlated Sources 

Let Afi and X2 be alphabets of two correlated sources, where Afi and X2 may be countably 
infinite. Let (Xi,X2) = {(X]^,^^)}^^ denote a general correlated source pair, i.e., (X",X2) is 
a pair of correlated (source) variables of block length n, and we write as 

(X", X2) = (Xii, X21), (X12, X22), • • • , {Xin, X2n), 

and let xj = Xji,Xj2, ■ ■ ■ ,Xjn be a realization of random variable X" (j = 1,2). The probability 
distribution of (xi,X2) is denoted by Pxi^x^ {^1,^2)- In particular, in the case that the pair of 
correlated sources has an i.i.d. property, it holds that 

n 

-PxrX2"(xi,X2) = Y[Px^X2{xii,X2i), 

i=l 

with generic correlated random variable {Xi,X2). 

B. e-Achievahle Rate Region 

The fixed- length codes for correlated sources are characterized by a pair of encoders ((/f)n \ 
and a decoder -^n- The encoders are mappings such as (l)n'' '■ — t- Ain \ (pn'^ '■ ^2 ~^ -^nK where 

denote the code sets. The decoder is defined as a mapping ^pn : Mn^ x Mn'' x X2- In the 

sequel, we focus on this Slepian-Wolf type source coding problem. 

The performance of fixed-length codes is evaluated in terms of the error probability and the 
codeword length. The error probability is given by 

e„ = Vt{{X^,X-) + ^.(</>W(Xn,0i')(X^'))}. 

We call such a pair of encoders {4>n\(j)k'*) and decoder il^n along with error probability an 
(n,MP,MP,e„) code. 

Definition 2.1: A rate pair i?2) is called an e-achievable rate pair if there exists an (n, Mn \ M^\en) 
code satisfying 

hm sup — log < Ri and hm sup — log M^^^ < R2 , 

n— ^00 ri Yi^QQ Tl 

limsupe^ < £• 

n— ^00 
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Then, the e-achievable rate region is defined as the set of all e-achievable rate pairs: 
Definition 2.2: (e- Achievable Rate Region) 

i?(e|Xi,X2) = {{R\,R'2)\{R\^R2) is e-achievable rate}. 
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For general correlated sources, Miyake and Kanaya |15j with finite source alphabets have de- 
termined the 0-achievable rate region and Han [8] with countably infinite source alphabets have 
established the e-achievable rate region for < Ve < 1. Before describing their results, we need to 
define the spectral sup-entropy rate as follow^. 

i^(Xi|X2) = p- lim sup — log 
i/(X2|Xi) = p- lim sup — log 

n— ^oo n. 

i^(XiX2) = p- lim sup — log 

n— ^oo n. 

Assuming that the pair of correlated sources has an i.i.d. property, it holds that 

F(Xi|X2) = H{Xi\X2), :H^(X2|Xi) = HiX2\Xi), HiX.X^) = H{X^X2), 

where H{Xi\X2) denotes a conditional entropy and H{XiX2) denotes a joint entropy. 

The following first- order achievable rate theorem reveals the rate region for general correlated 
sources. 

Theorem 2.1 (Miyake and Kanaya fl^): For any general sources with finite alphabets, the 0- 
achievable rate region is given as the set of (i?i,i?2) satisfying 

i?(0|Xi,X2) = {(i?l,«2) \Rl > i7(Xi|X2),i?2 > ^(X2|Xi), Ri+R2> F(XiX2).}. 

Han [H] has generalized their theorem to the case < e < 1 with countably infinite alphabets; set 
the function Fn{Ri, R2) as 

F^{R„R2) = Pr I i log ^ > ^1 °r i log I . > R2 

or - log ^—^^ — — > Ri+ R2 

then we have: 

^For any sequence {Zn}^=i of real-valued random variables, we define the limit superior in probability of {Z„}'^^i 
by p-limsup„^^ Zn = inf {/3| lim„_,oo Pr{Z„ > 13} = 0} (cf.[8]) . 
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Theorem 2.2 (Han JBi)'- For any general sources with countably infinite alphabets, the first- order 
e-achievable rate region is given by 

i2(e|Xi,X2) = CI Q(i?i,i?2) hmsupF„(i?i,i?2) ^ ^}) ' (2.1) 

where the function Cl(-) denotes the closure operation. 

This theorem has been established using the following key lemmas: 

Lemma 2.1 (Han 13^): Let M^^ and M^^ be arbitrarily given positive integers. Then, for all 
n = 1, 2, • • • , there exists an (n, Mn \ e^) code satisfying 

en < ^"i: \ ZnPxV^\X]}{^l\^2) ^ m ZnPxJ}\X'^{X2\Xl) < 7^ 

I Mi:' M\' 

where {zn}'^=\ is a sequence of arbitrary numbers such that Zi > (Vz = 1, 2, • • • ). ■ 

Lemma 2.2 (Han ]3I): Any (n, M^^\ M^^\ e„) code satisfies 



Er, > Pr|pxr|xj(^ri^?) < ^ or Pxj|xr(^?l^r) < 



orP.,.,(^r,X2"),-^}-3.., 



for all n = 1,2,-- - , where {zn}'^=i is a sequence of an arbitrary numbers such that Zi > 
(Vi = l,2,---)." 

Notice here that these lemmas are valid for general correlated sources with countably infinite 
alphabets. In this paper too we shall invoke these lemmas in order to derive the second-order 
achievable rate region as in the subsequent sections. 

C. {ai,a2,£)- Achievable Rate Region 

In the single source coding problem, a finer evaluation called the second-order achievable rate 
has been studied. Accordingly, we define the second-order achievable rate pair as follows. 

Definition 2.3: A rate pair (^1,^2) is called a second-order (oi, 02, e)-achievable rate pair if 
there exists an {n, Mi^\ Sn) code satisfying 

hm sup —= log — — - < Li and lim sup log — — - < L2 , 

lim sup < e. 



Notice that, in the definition of (oi, 02, e)-achievable rate pairs, the condition for the error proba- 
biUty is as same as the one in the definition of e-achievable rate pair. Moreover, let us define the 
set of (ai, 02, e)-achievable rate pairs given (01,02). 

Definition 2.4 (second-order {ai,a2,£)-achievable rate region): 

L(ai,a2,e|Xi,X2) = {(^1,^2)1(^1,^2) is (oi, 02, e)-achievable}. 

III. (ai, 02, e)- ACHIEVABLE RATE REGION FOR GENERAL CORRELATED SOURCES 

We shah determine the (ai, 02, e)-achievable rate region for general correlated sources with count- 
ably infinite alphabets. Before describing our main result, let us define the function Fn(Li, L2|ai, 02) 

by 

Fn{Li,L2\ai,a2) 

r - log Pxr I X? (^r I ^2 ) - ^01 - log Px^ I xr (^2 1 ) - ^02 ^ , 

rr <j -= > Li or ;= > L2 

'n \ n 



~ log Px^x- (^r^?) - n (ai + a2) ^ ^ , ^ 

or ^—^ — >Li+L2f, 

In 



and set 

Fn{Lx,L2\ax,a2) 

= \- Fn{Li,L2\ai,a2) 

(-logPx^lxs{Xl^\X^)-nai - log P^jlxr (^2 I^D " ""2 , 
— rr < — < Li, — < L2, 



logPxrxj(XrX2-)-n(ai+a2) ^ , ^ 

< Ivl + iv2 



m 

It should be remarked that the function thus defined Fn{Li, L2|ai, 02) is a multivariate cumulative 
distribution function. Now, we have 
Theorem 3.1: 

L(ai,a2,e|Xi,X2) = Cl(ULi,L2) limsup F„(Li, L2|ai, 02) < e M . (3.1) 

Proof: It is obvious by virtue of De Morgan's law (|3.ip in Theorem 13.11 is equivalent to 

L(ai,a2,e|Xi,X2) = Cl({(Li,L2)|HminfF„(Li,L2|ai,a2) > ^^"^^ 

so that in the sequel we will give the proof to p.2p instead of (|3.ip . 
1) Direct Part: 
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1 m1') 

lim sup — = loe < L2 , 



For any fixed (^1,^2) satisfying 

liminf F„(Li,L2|ai,a2) > 1 - e, (3.3) 

n—^oc 

set 

71^(1) _ nai + LiV^+2-$/n7 /\#(2) _ na2 + -L2 v^7 

wliere 7 > is an arbitrary small constant. It is obvious that 
and 



hold. Thus, in this direct part it suffices to show the existence of an {n, Mj]^\ e^) code such 
that limsup„^^e„ < s. 

Lemma \2A\ implies that there exists an {n,Mn\Mn\€n) code such that 

„ [ 1 , 1 ^ nai + Liv^ + 2^7 

= rr < — = log ^ ; > ;= , 

^J_j 1 ^ na2 + L2v^ + 2^7 

1 , 1 ^ n(ai+a2) + (Li + L2)V^+4^7\ 

Z„/-Xf -^2 j V'^ J 

Since {^^nj^JLi is a sequence of arbitrary numbers satisfying Zi > (Vi = 1,2, • • • ), we set Zn = 
g-v^7_ Then, we have 
. <Prl ^ lor ^ ^ nai + LiV^ + 2^7 ^7 

Iv^ ^Pxrix,"(^rl^2") " V^' 

1 , 1 ^ na2 + iv2\/n + 2v^7 v^7 
or log ; > := — , 

^Pxjixr(^2"l^r) " 

nr ^ In-. ^ ^ n(ai + Os) + (Li + L2)v/^ + 4^7 ^7 ] 4/^^ 



Hence, 



Sr,. < Pr 



logPx^lx-{X^\X^)-nai^^ 



-l0gPx,"|Xr(^2l^r)-^02 ^ ^ , 7 

or ^ >L2 + 



or 



log Px^x^ (X^X^) - n(ai + aa) 



n 



>Li + L2 + 4^!>+3e-^'^ 



< Pr 



logPx^\x-{X^\X^)-nar j 

1= > ^1 + 

'n \ n 



- log Px?|Xr (^2l^r) - ^ , , 7 

or p > ^2 + 



or ^— -j= > Li + L2 + 



+ 3e" 



■ ^7 



Hence, by means of De Morgan's law, we have 

logPxr|xj(^rl^2")-^ai 



n 



7 



n 



- log PxjIXf (^2"l^r) - ^ , , 7 

or ^ > -^^2 + 



n 



n 



-logPxpxH^r^2)-'^(ai+«2) ^ , , 27 

or ^ > i-i + ^2 + -7^ 

/n \ n 



=1-Pr 



-logPxr|xj(^rl^2")-?^ai , 7 
< i^i + 



- log Px^ I xr (^2 I ) - ^ 7 

In \/n 



logPx^x-{X]'X^)-n{ai+a2) ^ ^ 27 

y= < Li+ L2 + 

In \ n 



ai,a2 



Substituting ()3.5p into (|3.4p . we have 

'01,02^ +3e-^T 
< l-F„(Li,L2| 01,02) + 3e-^^. 
By taking limsup„_^^, we have 

limsupe„, < 1 — liminf (Li, L2I oi, 02) < e, 

n— >oo n— >-oo 

where the last inequality follows from <\3.3\i . Thus, the direct part has been proved. 



2) Converse Part: 
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Suppose that a pair (Li,L2) such that 

Ummf F,i(Li,L2|ai,a2) < 1 - e, 

n— >-oo 

is (oi , 02 , e)-achievable. Then, from the assumption, for any smaU 7 > 0, we have a code (n, Mn'' , Mn^'' , 
such that 

1 Mil) 

<Li+j, (3.6) 



1 


log 




Vn 




1 


log 




Jn 





<i^2+7, (3.7) 
for all sufficiently large n, and that 

limsupe„ < e. (3-8) 

n— ^00 

Thus, substituting (I3.6P and (13.7P into Lemma 12.2^ the error probability is lower bounded by 

r-iogPxrixH^ri^2)-nai logz, 

> Pr <^ ^ > -^^1 + 7 



-logPx2"|Xf (^2"!^") - ^^^2 ^ r logz, 
or ' > iy2 + 7 



n \/n 



- log Px^x^ jX^X^) - n(Qi + 02) ^ , , , , , logz,. . ^ 
or ^ > Li + L2 + 27 — } - 3zn, 



for all n = 1,2,-- - , where {zn}^=i is a sequence of an arbitrary numbers satisfying > 
(Vi = 1, 2, • • • ). Set Zn = e~^'^ and substituting it into the above, we have 

en > Pr <^ ^ > Li + 27 

\/n 



- log PxjIXr (^2l^r) - ^ , , ^ 

or — —= > ^2 + 27 



or 



'n 

\ogPx^x-{X^Xl^) - n(ai + 02) 



n 



>Li + L2 + 37I - 3e-^^ 



-logPx"lx-(^rl^2) -^^al 
> Pr <! ^ V 11 2; 1 > + 27 



or ' > ^2 + 27 
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Here, again, owing to De Morgan's law, we have 

r-logPx"|X"(^rl^2') -^oi 
\/n 



^ ^2 1^1 V 21 ij 2 ^ ^ 27, 



n 



log Px"X"(^r^2)- ^(01 + 02) 1 ^ 



= l-F„(Li + 27,L2 + 27|ai,a2)-3e-^^. 
By taking limsup^_^(^, from (|3.8p we have 

e > hmsupSn > 1 — liniinf F„ (Li + 27, L2 + 27I ai, 02) . 

n— >oo Ti— >oo 

Since 7 > is arbitrary, this means that 

L(ai,a2,e|Xi,X2) G Cl(<^ (Li,L2) liminf F„,(Li, L2|ai, 02) > 1 - e> 
Thus, we have proved the converse part. ■ 

IV. (oi,a2,e)-ACHIEVABLE RATE REGION FOR I.I.D. CORRELATED SOURCES 

Although, Theorem 13.11 is valid for general correlated sources, it is hard to compute it. In the 
subsequent sections, we compute the (ai, 02, e)-achievable rate region for some illustrative cases. 
To do so, in this section we assume that the pair of correlated sources has an i.i.d. property. 

In this case. Theorem 12.11 reduces to the first-order Slepian-Wolf theorem for i.i.d. correlated 
sources, i.e., we have 

i?(0|Xi,X2) = {(i?l,i?2)|iil >i?(^l|X2),fl2 >i^(X2|Xi), Ri+R2>H{X^X2)}, 

with generic correlated source variable (Xi,X2), which forms the pentagon with the following 
boundary points: 
Case I (Corner Points): 

ai = H{X^\X2) and 02 = H{X2); 
ai = H[Xi) and a2 = H[X2\Xi), 



Case II (Non-Corner Points): For < VA < 1, 

ai = \H{Xi) + {l-\)H{Xi\X2) and a2 = {I - \)H{X2) + \H{X2\X{), 
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Case III (Full Side Points): 

ai = H{Xi\X2) and 02 > ^(^2); 
oi > H[Xi) and a2 = if(X2|Xi). 
In each of these cases, we compute the (ai, 02, e)-achievable rate region on the basis of Theorem 

EH 

Remark 4^.1: It should be noted here that it is not difficult to check that (Li,L2) can take 
arbitrary values in if (01,02) is an internal point of the pentagon; and Li = L2 = +00 if 
(01,02) is outside the pentagon. Thus, we may focus only on the above cases. 

First, we define 

( - \ogPx^\x^{X?\XV:) - nHiXi\X2) 
^Ti,T2,T3) = hm Pr ^ '""^ ^ ^ ' ' ^ ^ ' ' < Ti, 



logPx?|xr(^2"l^") - nH{X2\Xi) 

< I2 



In 

- log Px-,x^ {X^XJ^) - nH{X,X2) ^ ^ \ 
Vn J 

then, by means of the central limit theorem, we see that ^{Ti,T2,Ts) specifies three-dimensional 

normal cumulative distribution function; more specifically, 

where y = {yi,y2,y3) is a three-dimensional row vector, and S = {crfj) = 1,2,3) denotes the 
covariance matrix, which is given by 

'^iJ ^ XI XI PXiX2{xi,X2)Zi{xi,X2)Zj{xi,X2), 

where 

2;i(a;i,a;2) = log— \ — . — - - H{Xx\X2), 

t^Xx\X-2\X\\X2) 

Z2{XI,X2) = log— ■ - H{X2\Xi), 

PX2\XAX2\XI) 

Z3{X1,X2) = log— ] - H{XiX2). 

PX^XAX\,X2) 

We then consider the marginal cumulative distributions '^23 (^2 , Ta ) , <I>3(T3) of <I>(Ti, T2, T3) such 
that 

^23(T2,r3)= lim $(ri,r2,r3), 
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^3(^3)^^11111 $(ri,r2,r3). 

Ti ,T2— >oo 

Other marginal cumulative distributions can be defined as in similar manners. Notice that <I>3(T3) 
is a one-dimensional normal cumulative distribution function. 
Now, we have the following theorem: 

Theorem ^.1: For any i.i.d. correlated sources, we have for all < e < 1: 
Case I ai = H{Xi\X2) and 02 = H{X2) (without loss of generality): 

L(ai,a2,e|Xi,X2) = {(^1,^2) |$i3 (Li,Li +L2) >l~e}. 

Case II (for all < A < 1): 

L(ai,a2,e|Xi,X2) = {(Li, L2) |$3 (^i + ^2) > 1 - e} ■ 

Case III ai = H{Xi\X2) and 02 > H{X2) (without loss of generality): 

L(ai,a2,e|Xi,X2) = {(Li, L2) |$i (Li) > 1 - e } . 

Remark 4-2: Notice that both of <I>3 (Li + L2) and <I>i (Li) are one-dimensional normal cumu- 
lative distribution functions. Thus, they can be simply written as 

1 f yl 



$3 (Li + L2)= / exp --^ d2/3, 

J -00 V27rcr33 V 20-33/ 



and also 



dyi J ^^^z^iVHet^^^^ V^-^^^^^^^-^^V ' 



where 



c^ii cri3 



5^13 = „ „ , yi3 = iyiiVs)- 



•^31 "^33 

Thus, the second-order achievable rate region in case II reduces simply to Li + L2 > Tu where 
Til is specified by 



/_ 



exp ( -7732" My3 = 1 - e; 



V^(733 V 2(t|3^ 

and the second-order achievable rate region in Case III reduces simply to Li > Tm (L2 is arbitrary) 
where Tm is specified by 

exp ( -7TZ2~ ] ayi = l-e. 



'11 
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On the other hand, the second-order achievable rate region in Case I is exphcitly written as the 
set of all (Li, L2) such that 

dyi J dy3^—j=== exp f --yiaS^gVfs j > 1 - £• 

Remark 4-3: Notice that, Case III is substantially the same as the full side-information type 
problem. Actually, in Case III, there is no condition on L2 for the achievable rate region. This 
means that the error probability is independent on the second-order rate L2- This is because, for 
the decoder, the first-order rate 02 > H{X2) is sufficient to reconstruct = X2. Therefore, the 
decoder is able to decode X2 = X2 perfectly without the knowledge of X" = xi. Then, using 
X2 = X2 the decoder decodes X" = xi. That is, X2 = X2 can be regarded as providing full side- 
information. Consequently, the result of Case III coincides with the results in [13], |14j . in which 
full side-information type problems are treated. ■ 

Remark 4-4- Theorem 14. II can be restated as the following corollary. Let 

Kn{ai,a2,e\Xi,X2) 

= {(Li, L2) 1$ (v^(ai - H{Xi\X2)) + Li, V^(a2 - HiX2\X^)) + L2, 

V^{ai + a2-H{XiX2)) + Li+L2) > 1 - e}, (4.1) 

denote the set of pair of real numbers given ai, a2 and e. Then, we have: 

Corollary 4-1' For an i.i.d. correlated sources, the second-order (ai, 02, e)-achievable rate region 
is given as the set: 

L(ai,a2,e|Xi,X2) = lim i^r„(ai, 02, e|Xi, X2). 

n— >oo 

Proof: It is not difficult to verify that i^„(ai, 02, e|Xi, X2) asymptotically coincides the each 
case in the theorem, by letting oi and 02 be as in Case I, II and III, respectively. Moreover, the 
corollary also includes trivial cases such as (01,02) is inside or outside the pentagon (cf. Remark 
I4T]). ■ 

Proof of Theorem \4.1\ 
1) Case I: 
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In view of Theorem 13.11 in the form of ()3.2p , it suffices to calculate 
lim F„(Li,L2|ai,a2) 

n— >-oo 

(-logPx^lxs{X^\X^)-na, -log Px^\xr{X^\XY) - na^ ^ 
— iim rr < ;= < Li, — < L2, 



logPxrx,"(XrX2")-n(ai + a2) ^ , ^ 

< ivl + l72 



In 

Substituting a\ = H{Xi\X2) and 02 = H{X2) into F„(Li, L2|ai, 02), we have 
^"71(^1,-^^2 1 01,02) 

r-logPxr|xj(^r|X?)-^^^(^i|^2) , -logPxj|x,.(X2-|Xr)-nif(X2) ^ 

— rr < — < Li, — < Jv2, 

Jn Jn 



\ogPx^x-{X^Xl^)-n{H{Xr\X2) + H{X2)) ^ , ^ 



n 



^ . -logPxr|X5'(^rl^?)-^^^(^i|^2) , ~\ogPxs\xY{Xl^\X'l)-nH{X2) ^ 

rr <^ -= < Li, -= < i72, 

yjn Vn 



-\ogPx^x^i{X^Xli)-nH{X^X2) ^ , ^ 



n 



^ . -\ogPx^\x^{X^\X'^)-nH{X^\X2) ^ 
J^ 



\ogPx^\x-{X^\X'^) -nH{X2\Xi) ^ 

^ " ^' ^ " < v^(ff(X2)-g(X2|Xi)) + L2, 



logPxrx,"(^r^2") - nH{XiX2) 



<Li + L2} , 



fn 

where the second equality is derived by the chain rule, H{Xi\X2) + H{X2) = H{XiX2). Moreover, 

H[X2) — H{X2\Xi) = I[Xi;X2) > holds, since we are considering correlated sources. Thus, for 

any constant > 0, we have y/n{H(X2) — H{X2\Xi)) > W for sufficiently large n. 

As a consequence, noting that the correlated sources has an i.i.d. property, and taking account 

of the asymptotic normality (due to the central limit theorem), we have 

J. n ri , f -logPx^\x^{XlW2)-nH{X,\X2) 

F„(Li,L2 ai,a2 > Pr <^ j= <Li, 

Jn 



-\ogPxs\XY{X^\Xl^)- nH{X2\X,) 

< W + L2, 



/n 

- log Px^x^ jX^X^) - nH{XiX2) 
-)>$(Li,M^ + L2,Li +L2) (asn^oo). 



<-Ll +iv2 
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Furthermore, as W > can be arbitrarily large, this implies that 

limsupF„(Li,L2|ai,a2) > lim $ (Li, + Lg, Li + L2) = $13 (^i, + -^^2) • (4.2) 

n->-oo W^oo 

On the other hand, it is obvious that 

F„ Li,L2 ai,a2 < Pr <^ — < Li, 



so that we have 



■\ogPx-x^{X^Xl^)-nH{X^X2) ^ , ^ 



liminf F„(Li,L2|ai,a2) < ^13 (^1,^1+^2) . (4.3) 

n— ^00 



Thus, summarizing (|4.2p and (|4.3p leads to 

lim F„(Li,L2|ai,a2) = ^13 (-^^1,^1 + ^2) . 

n— ^00 

Therefore, the proof of Case I has been completed. 

2) Case II: 

First, notice that 

01 + 02 = \H{X^) + {l-\)H{X^\X2) + {l-\)H{X2) + \H{X2\X^) 
= A {H{Xi) + H{X2\Xr)) + (1 - A) {H{X^\X2) + ^(^2)) 
= H{XiX2). 

Moreover, Ai = oi — H{Xi\X2) > if Xi and X2 are correlated, because 

oi - H{X^\X2) = \H{X{) + (1 - A)F(Xi|X2) - H{Xi\X2) = A (i?(Xi) - H{Xi\X2)) > 0. 

The same argument yields, A2 = 02 — ^^(-^21-^^1) > 0, so that Fn,(Li, L2|ai, 02) is given by 

_ r -logPx"lX"(^rl^2) - f^^^(^ll^2) 
F„(Li,L2|ai,a2) = Pr ""^ ^ ^ ' ' ^ < AiV^ + L^, 



l0g^X,-|Xr(-^2"l^r) -»g(^2|Xi) 

^ < A2 + L2, 



logPx^X5iX^X^)-nH{X^X2) ^ , ^ 



/n 

Here, Ai^/n and A2-\/ra goes to 00 as n —t- 00. Therefore, again, by virtue of the asymptotic 
normality, it holds that 

lim FniLi,L2\ai,a2) = ^siLi + L2). 
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Therefore, the proof of Case II has been completed. 



3) Case III: 

Notice that, setting 6 = a2 — H[X2) > leads to 

ai + a2- H{XiX2) = H{X^\X2) + H{X2) + 5 ~ H{XiX2) 
= (5>0. 

Moreover, A2 = a2 — H{X2\Xi) > holds, because 

02 - H{X2\X^) = H{X2) + 5- H{X2\X^) > 0. 
holds. Thus, F„(Li, L2|ai, 02) is given by 

F„(Li,L2 01,02) = Pr<^ ^ — 2 j= < Ll, 



-logPxs\XY{X^2m-nH{X2\X,) 

1= < ^2V^ + ^2, 

- log Pxrx,- [X^Xlj) - nH{X^X2) ^ ^ \ 

-= < ^^/n + Ll + L2 > , 

Vn J 

Here, A2\/n and 6y/n goes to 00 as n —t- 00. Then, again, by virtue of the asymptotic normality 
as well as the same discussion in Case I and II, it holds that 

lim F„(Li,L2|ai,a2) = $i(Li), 

n— >-oo 

thus completing the proof. ■ 
Remark 4-5: It is interesting to note that in Case II and III, the condition on (ai, 02, e)-achievable 

rate region is described by one-dimensional normal distribution function, while in Case I that is 

described by two-dimensional normal distribution. 

Remark 4-6: It should be emphasized that the computation in this section is on the basis of the 

asymptotic normality. Hence, this approach is valid not only for i.i.d. correlated sources but also 

for sources whose self-information vector has the multi-dimensional asymptotic normality. 

V. (ai, 02, e)- ACHIEVABLE RATE REGION FOR MIXED CORRELATED SOURCES 

In the previous section, we see that the second-order achievable rate region of any i.i.d. corre- 
lated sources (or more generally, correlated sources whose self-information vector has the multi- 
dimensional asymptotic normality) is described by using the multi-dimensional normal distribution. 
In this section, we try to compute the second-order achievable rate region of the correlated sources 
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in which the asymptotic normahty of self-information vector does not hold. One of such source 
classes is the mixed correlated sources. Recall that the mixed sources are typical cases of nonergodic 
sources. We consider two typical cases of mixed correlated sources and determine the second-order 
achievable rate region in each case explicitly by virtue of the relevant multi-dimensional asymptotic 
normality. 

A. Mixture of Countahly Infinite Correlated Sources 

In this subsection, we assume that the correlated sources is a mixture of countably infinite i.i.d. 
correlated sources. Notice that 

Let (A; = 1,2,---) be arbitrary pairs of i.i.d. correlated sources indexed by k. The 

mixed correlated sources that we consider in this subsection is defined by 

-PXi"X2"(xi,X2) = ^'w(/c)P^(fc)r.^(fc)„(xi,X2), (5.1) 

k = l 

where w{k) > (A; = 1, 2, • • • ) are constants such that XlfcLi w{k) = 1- 

The following lemma, which is implicitly contained in Han [8], plays the key role. We give the 
proof in the appendix for the sake of reader's convenience. 

Lemma 5.1: Let {zn''}'^=i, {^n''}'^=\, {^n''}'^=i be any real-valued sequences. Then for the 
mixed correlated sources defined by (|5.ip . it holds that, for /c = 1,2, • • • with w{k) > 

1) 



Pr < 



n \ n 



< z. 



(3) 



n 



< Pr < 



-logP^(.)„ (Xf ^n^f ^'^) -logP^,.) 



n Wn 



n w{k) 
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2) 



Pr ( 



< z. 



(1) 



n 



< z, 



(2) 



n 



< z. 



(3) 



n 



> Pr < 



n 



log P^(fc)n|^(A:)n 



^(fc)n|^(fc)n 



< z, 



(2). 



n 



n 



where 7^ > satisfies 71 > 72 > • • • > 0, 7n — 0, \/n7n ~^ 00. 
Proof: See Appendix. ■ 
Moreover, we define tlie normal cumulative distribution function for A; = 1, 2, • • • , as follows. 



$W(ri,r2,r3) = I ' dyi [ ' dy2 [ ' dys 



(v^)3VdetS^ 



exp ( --yS, ) , 



where y = {yi,y2,y3) is a three-dimensional row vector, and = {crfj{k)) = l,2,3,fc 



1, 2, • • • ) denotes the covariance matrix, which is given by 



where 



xieXi X2&X2 
zf\x„X2) = log 

.f(x„X2)=l0g 
.f(x„X2)=l0g 



1 


p^(fc) 


^(fc)(xi|x2) 




1 




^(fc)(a;2|xi) 




1 




(fc)(a;i,X2) 
^2 



FX 



In addition, we set 

/Cr"(«i,a2,e|Xi,X2) 



(^1,^2 



fc=i 



5] ( (ai-iJ (Xf ) ) (a2-i? (xf |Xp)) ) +L2, 



n(ai + a2-//(xj''^xf^)) +L1+L2) >l-e}>, (5.2) 
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given ai,a2 and e. Then, the following theorem holds. 

Theorem 5.1: For a mixed correlated sources defined by (15. ip . the second-order (oi, 02, e)-achievable 
rate region is given as the set: 

L(ai,a2,e|Xi,X2)= lim ir--(ai, 02, e|Xi, X2). 

n— >cxD 

The theorem denotes that the second-order achievable rate region is given as the set defined by 
using the mixed three-dimensional normal distribution, while in the previous section the set is 
defined by using the single three-dimensional normal distribution. 

Remark 5.1: It should be noted that the case, in which ai ^ (A; = 1,2,---), 



02 ^ H [X^^^X^^^j (A: = 1,2,---) and ai + 02 ^ (^xf ^ j (A; = 1,2,---) hold, necessarily 
implies that 

k=l 

where / is some integer. And then, the second-order achievable rate region is trivial, that is, (Li, L2) 
can take arbitrary values in M? or Li = L2 = +00, depending on w{k) and e. On the other hand, 
we can also see that in nontrivial case, the second-order rate region is determined by using the 
mixture of two or one-dimensional normal distributions (cf. Remark 14.21 and Example 15. ip . 
Proof of Theorem I5.il ' From Theorem 13. H it suffices to show that 

lim F„(Li,L2|ai,a2) 

n— ^-oo 

00 ^ 

= i™, E ^(^)^^'^ ( (xf )|xf )) +Li, [a2-H +L2, 

V^(^ai+a2-H (xf ))) + Li + L2^ . 
From the definition of mixed correlated sources and the first inequality of Lemma 15. we have 
lim inf F„ (Li , L2 1 ai , 02 ) 

n— >cxD 

- f-logPxr|xj (Xf )'^|xf") -na, -logP^ji^r (xf"|xf -na2 
= liminfyu;(A;)Pr<^ ^ r- ' <Li, ^ ^ ^ <L2, 

n— >oo ^ — ' I -y^Jl 

- log Pxrxj (Xf )"xf )'^) - n(ai + 02) 

^ <Li+L2 



dog P^{fc)Ti| „(fc)n ( x|'^^"'|X2'^''" ) — noi 



k=l 



w(k) liminf I Pr < ;= <Li+^. 
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-log i-'^(fc)„|^(fc)„ I X2'^'"''|X|" 



na2 



n 



<L2 + ln 



n{ai + 02) 



n 



< Li + L2 + 7n > + 



w{k) 



((k) (k)\ 
XI , ^2 ) [k 

1, 2, • • • ) and similar discussion with the proof of Theorem 14. H it holds that 
lim inf F„(Li, L2|ai, 02) 

00 / 

< liminf ^u;(A;)$(*^)( v^(^ai-i7 (^x['"^\X^'"'^^^+L^,^/^ (^a2-H (^xf +L2, 

y^(ai+a2-H (xj^^xf ))) + Li + L2 ) . 



On the other hand, the lower bound of liminf„^oo -P'n(^ij -^2|oi; ^2) can be shown similarly 
with the argument above, where the second inequality of Lemma 15.11 is used instead of the first 
inequality of Lemma 15. 1[ ■ 

Remark 5.2: As shown in the above, the analysis here for mixed correlated sources is on the 
basis of the asymptotic normality of self-information vector and Lemma 15.11 This means that the 
similar argument is valid for any mixture of countably infinite sources for which the asymptotic 
normality of self-information vector holds for each of the component correlated sources. 

Example 5.1: Let us consider the mixed correlated sources, for which it holds that w{l) + w{2) = 
1, and 



H{X. 



> H X. 



-(2) 



Here, let us compute the second-order achievable rate region: 



and 



L{H{x['^\xi'A,H(xi'^),e 



Xi, X2 



Xi,X2 . 



Notice that the second-order achievable rate region depends on w{l) and e. From Theorem 15.11 
and Remark 14.11 it is easy to verify that if w{l) > e, we have 



L{ H ixi^'^lxi'"'^ 



,H [X. 



'(1) 



Xi, X2 



w{l)^^^l(^Li,Lr + + w{2) > l-e|, 
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L[H[x['^\xi'^),H(x^'U,e 



Xi,X2) = (+00, +00) 



if w(l) < e, we have 



H{xi'U,e 



L[H[X^^^\xi'U,H[X^-'\,e 



'(2) 



{Li,L2) 



u;(2)<I>g(^Li,Li+L2^ > 1 



and otherwise (i.e., w{l) = e) we have 



L{H {X[^^\X^2^^ 



Xi, X2 



Xi,X2) = (+00, +00). 



B. General Mixture of i.i.d. Correlated Sources 

In this subsection, we consider an extension of Theorem 15. II to the case general mixture instead 
of countably infinite mixture. 

The correlated mixed source that we consider in this subsection is defined by 



-PxfX|'(xi,X2) = / P^(9)n^(e)„(xi,X2)w(d6'), 
J A ^ ^ 



(5.3) 



where w{dO) is an arbitrary probability measure on the parameter space A, and ^X^^'' , X2^^^ [6 E A) 

are i.i.d. correlated sources with finite alphabet. 

We use the following lemma shown by Han [S] Remark 7.4.2], instead of Lemma |5. 11 

Lemma 5.2 (Han JS^): Let {zi^^}^=i, {z^n^}^=i, {z^n^}n=i be any real-valued sequences. Then, 

for the mixed correlated sources defined by (|5.3p . it holds that. 



lim inf Pr 

n— >oo 



-log P 



-logP^{s)„|^(e)„ (X2'''"'|X| 



n 



n 



-logP^(«)„^(«)„ ^ ^ 



n 



< lim inf / Pr 

J A 



< z. 



-logP,„.,(xflxr 



n 



n 



< 4'^ ) wide) 



n 
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< limsup / Pr { ^' ^ y^^'J < ,(1)^ ^' ^ I ^ ' ^ / < ,(2) 



71— >oo J A 



n \ n 



n 



< J lim sup Pr <! ' < z\'> +7„, 



'A n^oo 



n 



n Wn 



where 7n > satisfies 7i > 72 > • • • > 0, 7„ — > 0, \fn^n 00. 

In this case, we also define the normal cumulative distribution function for £ A as follows. 

fTi /•T2 i-Ts / Y 



J —00 J —00 J —{ 



wherey = (2/1,^21^3) (a three-dimensional row vector), and the covariance matrix = {crfj{0)) 
1, 2, 3, ^ E A) are defined as the similar manner with the previous subsection. 
Finally, we set 

v^(ai +02 -i? (xf^xf))) +Li+L2^w{d0) > l-e|, (5.4) 

given ai,a2 and e. Then, the following theorem holds. 

Theorem 5.2: For a mixed correlated sources with finite alphabet defined by (|5.3p . second-order 
(oi, 02, e)-achievable rate region is given as the set: 



L(ai,a2,e|Xi,X2) = lim /^^(ai, 02, e|Xi, X2). 

n— >cxD 



Proof: It suffices to proceed in parallel with the arguments as made in the proof of Theorem 
15.11 Notice that Lemma 15.21 is used instead of Lemma 15.11 ■ 



VI. Concluding Remarks 

So far we have established the second-order source coding theorem for the Slepian-Wolf coding 
problem with general correlated sources. On the other hand, in single-user source coding problem. 
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Hayashi [7] has shown the second-order opthnal achievable rate for general sources, and computed 
it for an i.i.d. source using the asymptotic normality. In the case that the source is assumed to 
be in the class of mixed sources, Nomura and Han [S] have computed the second-order achievable 
rates explicitly. 

Analogously, we have established the second-order achievable rate region for general correlated 
sources, and computed it for i.i.d. correlated sources and mixed correlated sources, respectively. 
As we have mentioned in the above, in order to compute the achievable rate region for i.i.d. 
correlated sources, the multivariate normal distribution played the key role. In particular, the 
condition for the second-order achievable rate region turned out to be expressed in terms of the 
marginal distributions of a three-dimensional normal distribution. 

Notice here that, in [9j the second-order optimal rates for mixed sources in single- user source 
coding problem has also been denoted by the form of the equation, which is similarly to (|5.2|) and 
(|5.4p . While the equation in [9j is based on the single-normal distribution functions, the similar 
formula in the present paper is based on the multi- normal distribution functions. In this viewpoint, 
results in the present paper can be considered as a generalization of results in [9j. In the channel 
coding problem, Polyanskiy, Poor and Verdii [18] have derived the second-order optimal capacity 
rates for Gilbert-Elliott channel by using the similar form of the equation. 

Finally, it should be emphasized that our analysis is on the basis of the information spectrum 
methods and hence our results in this paper are valid with countably infinite alphabet excepting 
Theorem 15. 2[ 



Appendix 
Proof of Lemma 15.11 

We first show the first inequality. Set a sequence {7n}^i satisfying 71 > 72 > • • • > 0, 7„ — )■ 0, 
y/n'fn — ^ 00 and define three sets 



Z?(2)(A;) = J(xi,X2)G^2"xA'i" 



-logPxrX2"(xiX2) -logP^(fc)„^(fc)r.(xiX2) 



n 



(Xi|x2) -logP^(.).|^(.).(xi|x2) 




for A; = 1, 2, • • • . In addition, we set 

D^ik) = Di^\k)UD^^\k)UDi''\k). 
Then, it holds that for A; = 1, 2, • • • 



(xi,x2)eDj.'^\fe) 



(xi,x2)eDi'^>(fe) 



< e" 



-Vn'Tn 



Similarly, it holds that 



(xi,x2)eDj,''(fc) 



(xi|x2)P^(fc)n(x2) 



(xi,X2)eDl''(fc) 

< J] Pxrx2"(xi,X2)P^(.)„(x2)e-^'^", 
(xi,x2)eDi''(fc) 

for /c = 1,2,-- - . 

Notice here that from the definition of the mixed correlated sources, we have 

oc 

PX^{X2) = y^Py(>=)r.(x2)t<j(fc), 

k=l 

for A; = 1, 2, • • • . Thus, we have 

P (^^< ^^2(^2) 
(fc)„(X2j S 77^' 

for k = 1,2, ■ ■ ■ with ?x;(A;) > 0. Substituting the above inequality into ()A.2p it holds that 
Pr{(xf e < J] Pxr|xj(xi,X2)Px2"(x2^'"'^'" 



e 

< 



(xi,x2)eD^i''(fc) 



i(;(A;) 

for A; = 1, 2, • • • with w{k) > 0. Similarly, we have 

PT{(x['^-,xi'^-)eD(^\k)}<'-^, 

for A; = 1, 2, • • • with w{k) > 0. 
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From the definition of Dn{k), by means of union bounds, (jA.ip . ()A.3P and (IA.4P yield 



Pr{(x( 



{k)n ^{k)n I ^ 



' ' w[k) w[k) 



for A; = 1, 2, • • • with w{k) > 0, because < w{k) < 1 holds. 
This means that 



Pr < 



> 



n 



n 



7r! 



logPx„l^„ (Xfn^f ^" 



Inn- P / VC^)" 



> 



> 



logF^(fc)„l^{fc)„ (X^ \Xl 



n 



logP 



^{k)n ^(k)n 



n 



n 



> 1 



w{k) 



Hence, we have 



Pr 



logP^,.)„ (Xf -logP^(.)„,^(.)„ (X^'^lxf)'^ 

^-7n <4'\ — 



n 



7n < 



— log P^(fc)„^(fc)„ [X^ X2 



^(k)n -tr{k)n 



n 



In < Z, 



(3) 



> Pr 



-logPxrixj fe'^l^f" 



n 



< z, 



< z. 



(3) 



(1) 



logPxjIXr 1^1'^" 



(2) 



n 



w{k) 



for /c = 1, 2, • • • with 'u;(A;) > 0, which is the first inequality of the lemma. 
Next, we show the second inequality of the lemma. Set 



= {(xi,X2) G X |Pxr|x?(xi|x2) < e-^^"\ or Pxj|xr(x2|xi) < e'V^^-', 

or P_xrx^'(xi,X2) < e-^^"''| 
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5W = i(xi,X2)G^rX^2" 



P 



'(Xl|x2) < 



(1) 



-■s/nz 



(2) 



w{k) 



or P (fc)„| (fc)„(x2|xi) < — — 

^2 \^i ^ ' ' w{k) 



(3) 



or P (fc)„(xi,X2) < — - — > 

-^1 -^2 w[k) I 

From the property of mixed correlated sources, (xi,X2) G Si^^ {k = l,2,---) holds for (xi,X2) G 
Sn- This means that 

Q r- q{k) 



{k = 1, 2, • • • ). Moreover, since, by assumption, jn > — log^(fc) (^f^ = 1,2, ■ ■ ■ ) hold for sufficiently 
large n, we have 



< P 



Pr 



logP^(.)„l^(.)„ 



> z. 



(1) _ logw(A;) 



n 



n 



or 



or 



logP^(fc)„l^(fc)„ (X^ \Xl 



> z. 



(2) _ log'^(fe) 



n 



-logP^(.)„^(..). 



n 



> z. 



< 1-Ft< 



-log P 



n 



(3) _ log w{k) 
n 



n 



-10gP^(/c)„^(fc)n [X[^''"'X^^'^ 



n 



< - -Y 



logP^(fc)„|^(fc)„ (X^^'^^lxj''^" 



< Z, 



(2). 



n 



for sufficiently large n. 

Noting that, 1 — P„{fc)„„{fc)„(5„) is equal to the left hand side of the second inequality of the 
lemma, the second inequality has been obtained. ■ 
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